{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "db92dbb5",
   "metadata": {},
   "source": [
    "# 快速成为深度学习全栈工程师第10课书面作业\n",
    "\n",
    "学号：114499\n",
    "\n",
    "**作业内容：**  \n",
    "1. 相比于ResNet, ResNext最大的特点是什么？  \n",
    "2. 请实现BatchNorm, LayerNorm, InstanceNorm, GroupNorm（要求：都是2D normalization，都用pytorch实现）\n",
    "（提示：https://pytorch.org/docs/stable/nn.html#normalization-layers，实现了的normalization层，写出它的API即可，没有实现的，请基于BatchNorm2d实现）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f93692e8",
   "metadata": {},
   "source": [
    "## 第1题\n",
    "相比于ResNet, ResNext最大的特点是什么？  \n",
    "\n",
    "传统的深度神经网络要提高模型的准确率，都是加深或加宽网络，但是随着超参数数量的增加（比如channels数，filter size等等），网络设计的难度和计算开销也会增加。\n",
    "\n",
    "ResNext的思路是：\n",
    "* 增加基数 cardinality 比增加深度和宽度更有效。（在某一层并行transform的路径数提取为第三维度，称为基数，即”cardinality”。）\n",
    "* 用一种平行堆叠相同拓扑结构的blocks代替原来 ResNet 的三层卷积的block，在不明显增加参数量级的情况下提升了模型的准确率，同时由于拓扑结构相同，超参数也减少了，便于模型移植。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24aba38d",
   "metadata": {},
   "source": [
    "<img src=\"https://gitee.com/dotzhen/cloud-notes/raw/master/resnext.png\" alt=\"resnext\"  />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e143417e",
   "metadata": {},
   "source": [
    "上图所示提出了深度网络的新维度，除了深度、宽度（Channel数）外，作者将在某一层并行transform的路径数提取为第三维度，称为”cardinality”。跟Inception单元不同的是，这些并行路径均共享同一拓扑结构，而非精心设计的卷积核并联。除了并行相同的路径外，也添加了层与层间的shortcut connection。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99125584",
   "metadata": {},
   "source": [
    "## 第2题\n",
    "请实现BatchNorm, LayerNorm, InstanceNorm, GroupNorm（要求：都是2D normalization，都用pytorch实现） （提示：https://pytorch.org/docs/stable/nn.html#normalization-layers，实现了的normalization层，写出它的API即可，没有实现的，请基于BatchNorm2d实现）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "9cf1da31",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn.modules.batchnorm import _NormBase\n",
    "import numpy as np\n",
    "from torch import Tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb33bc65",
   "metadata": {},
   "source": [
    "预先定义一个向量用来后面验证："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "493f7a54",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[[ 0.4780,  0.3886, -1.1531, -1.1378,  1.3979],\n",
      "          [ 0.4481,  1.5727, -1.1699, -0.1222,  1.2142],\n",
      "          [ 0.3533, -1.0399, -1.5998, -1.7431, -0.1317],\n",
      "          [ 0.6088, -0.3355,  0.3167,  0.0123,  0.5800],\n",
      "          [-1.3356,  0.2547,  0.6722, -0.3275,  0.7841]],\n",
      "\n",
      "         [[ 0.0750, -1.7403,  2.1101, -0.2172,  0.9711],\n",
      "          [ 0.0078, -0.8967, -0.0840, -0.6134, -1.6067],\n",
      "          [-1.1052, -1.0673,  0.9192,  0.3653, -0.0255],\n",
      "          [ 1.1987, -0.3864, -0.1676,  0.0871, -0.4533],\n",
      "          [ 1.2827, -0.5812,  0.7017,  0.4895, -0.8983]],\n",
      "\n",
      "         [[ 2.0944, -0.0715,  0.0596, -0.5123, -0.4045],\n",
      "          [ 0.9442,  0.9312,  0.8480, -0.0750, -0.5964],\n",
      "          [ 0.5521, -0.7440, -0.4629, -0.0399,  1.6194],\n",
      "          [ 1.1379,  1.2512,  0.1351,  0.8222, -0.1243],\n",
      "          [ 0.7589,  0.7025, -0.3270,  0.7832, -0.7272]],\n",
      "\n",
      "         [[ 0.1059, -0.2193, -0.1878, -1.0491,  0.9100],\n",
      "          [ 0.4008,  2.4167,  0.1490, -1.1507,  0.0489],\n",
      "          [-0.5775,  0.3413,  0.0080,  0.0115, -0.1312],\n",
      "          [ 0.7722, -0.1959,  1.8806,  1.1583,  1.5807],\n",
      "          [ 0.0435,  0.1634, -0.2828,  0.1496,  1.2734]]],\n",
      "\n",
      "\n",
      "        [[[ 0.9942,  1.0359, -0.6319, -0.3279, -0.1802],\n",
      "          [-1.3898, -0.0690,  1.8222, -0.4009, -1.4214],\n",
      "          [-0.1150,  0.6664, -2.1495,  0.7807,  2.8578],\n",
      "          [ 0.3932, -0.2557, -0.4211, -1.6184, -1.6108],\n",
      "          [ 1.0972, -1.5090, -0.2528, -1.3030,  2.3881]],\n",
      "\n",
      "         [[-0.3443, -0.0074,  0.7016,  1.3449,  0.6968],\n",
      "          [ 0.2303, -1.4523, -0.8812,  0.1695,  1.2524],\n",
      "          [ 0.0769, -1.2696,  0.2851,  0.3548, -1.0442],\n",
      "          [-0.4578, -1.4943,  1.0638, -0.4481, -0.8379],\n",
      "          [-2.2977, -0.2837,  0.1902,  0.5277, -0.6822]],\n",
      "\n",
      "         [[ 1.0332,  1.5724,  1.1665, -1.1410, -0.2549],\n",
      "          [-0.2556,  1.6239,  0.0695,  0.1612, -0.9199],\n",
      "          [-0.1683,  0.5158,  0.2898, -1.0610,  1.1225],\n",
      "          [ 0.5960, -0.5939, -0.3263,  2.0311, -0.2143],\n",
      "          [ 2.2880, -1.0880, -0.6773,  1.5384,  0.5256]],\n",
      "\n",
      "         [[ 0.6005, -0.2254, -0.2668,  2.6472,  1.3369],\n",
      "          [ 1.2879, -0.8701, -1.4396,  0.1452,  0.2164],\n",
      "          [-0.4326, -2.3298,  0.8970,  0.4260, -0.2614],\n",
      "          [ 0.7445, -0.8680, -0.7984, -1.9995,  0.1516],\n",
      "          [-0.2673, -0.0463,  0.0104,  0.1806,  0.3793]]]])\n"
     ]
    }
   ],
   "source": [
    "input = torch.randn(2, 4, 5, 5)\n",
    "print(input)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93169ae5",
   "metadata": {},
   "source": [
    "### （1）BatchNorm实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "655ae7c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyBatchNorm(_NormBase):\n",
    "    def __init__(self,num_features, eps=1e-5,gamma=1., beta=0.):\n",
    "        super(MyBatchNorm, self).__init__(num_features,eps)\n",
    "        self.gamma = gamma\n",
    "        self.beta = beta\n",
    "\n",
    "    def forward(self, input: Tensor) -> Tensor:\n",
    "        # input_shape:[B, C, H, W]\n",
    "        B,C,H,W = input.shape\n",
    "        if C != self.num_features:\n",
    "            print(\"num_features don't match.\")\n",
    "            return\n",
    "    \n",
    "        x_mean = torch.mean(input,(0,2,3),True)\n",
    "        x_var = torch.var(input,(0,2,3),keepdim=True)\n",
    "        x_normalized = (input - x_mean) / torch.sqrt(x_var + self.eps)\n",
    "        results = self.gamma * x_normalized + self.beta\n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "689cf99a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[[ 0.4735,  0.3937, -0.9817, -0.9680,  1.2941],\n",
      "          [ 0.4468,  1.4501, -0.9967, -0.0620,  1.1302],\n",
      "          [ 0.3622, -0.8807, -1.3802, -1.5081, -0.0705],\n",
      "          [ 0.5901, -0.2523,  0.3295,  0.0580,  0.5645],\n",
      "          [-1.1445,  0.2743,  0.6467, -0.2451,  0.7465]],\n",
      "\n",
      "         [[ 0.2182, -1.7633,  2.4394, -0.1008,  1.1962],\n",
      "          [ 0.1448, -0.8424,  0.0445, -0.5333, -1.6174],\n",
      "          [-1.0701, -1.0287,  1.1396,  0.5349,  0.1084],\n",
      "          [ 1.4446, -0.2855, -0.0467,  0.2313, -0.3586],\n",
      "          [ 1.5363, -0.4981,  0.9021,  0.6705, -0.8442]],\n",
      "\n",
      "         [[ 1.9804, -0.4476, -0.3007, -0.9417, -0.8208],\n",
      "          [ 0.6910,  0.6764,  0.5832, -0.4515, -1.0360],\n",
      "          [ 0.2515, -1.2015, -0.8863, -0.4121,  1.4480],\n",
      "          [ 0.9082,  1.0352, -0.2160,  0.5543, -0.5068],\n",
      "          [ 0.4833,  0.4201, -0.7340,  0.5105, -1.1826]],\n",
      "\n",
      "         [[-0.0321, -0.3699, -0.3372, -1.2321,  0.8034],\n",
      "          [ 0.2744,  2.3689,  0.0128, -1.3377, -0.0913],\n",
      "          [-0.7421,  0.2125, -0.1338, -0.1302, -0.2784],\n",
      "          [ 0.6602, -0.3457,  1.8119,  1.0614,  1.5003],\n",
      "          [-0.0969,  0.0277, -0.4360,  0.0133,  1.1810]]],\n",
      "\n",
      "\n",
      "        [[[ 0.9340,  0.9712, -0.5168, -0.2455, -0.1137],\n",
      "          [-1.1928, -0.0146,  1.6727, -0.3106, -1.2211],\n",
      "          [-0.0555,  0.6415, -1.8706,  0.7435,  2.5966],\n",
      "          [ 0.3978, -0.1811, -0.3287, -1.3969, -1.3900],\n",
      "          [ 1.0259, -1.2992, -0.1785, -1.1155,  2.1775]],\n",
      "\n",
      "         [[-0.2395,  0.1282,  0.9021,  1.6042,  0.8968],\n",
      "          [ 0.3877, -1.4489, -0.8256,  0.3213,  1.5032],\n",
      "          [ 0.2202, -1.2495,  0.4475,  0.5235, -1.0035],\n",
      "          [-0.3635, -1.4948,  1.2974, -0.3528, -0.7783],\n",
      "          [-2.3717, -0.1734,  0.3439,  0.7122, -0.6084]],\n",
      "\n",
      "         [[ 0.7908,  1.3953,  0.9402, -1.6466, -0.6532],\n",
      "          [-0.6540,  1.4530, -0.2896, -0.1867, -1.3986],\n",
      "          [-0.5561,  0.2108, -0.0426, -1.5569,  0.8909],\n",
      "          [ 0.3007, -1.0332, -0.7332,  1.9095, -0.6076],\n",
      "          [ 2.1975, -1.5871, -1.1266,  1.3572,  0.2218]],\n",
      "\n",
      "         [[ 0.4819, -0.3763, -0.4194,  2.6084,  1.2469],\n",
      "          [ 1.1961, -1.0461, -1.6379,  0.0088,  0.0828],\n",
      "          [-0.5915, -2.5627,  0.7899,  0.3005, -0.4137],\n",
      "          [ 0.6314, -1.0440, -0.9716, -2.2196,  0.0154],\n",
      "          [-0.4199, -0.1902, -0.1313,  0.0456,  0.2520]]]])\n"
     ]
    }
   ],
   "source": [
    "m = MyBatchNorm(4)\n",
    "output = m(input)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b11ec6",
   "metadata": {},
   "source": [
    "### （2）LayerNorm实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "8a6df4c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyLayerNorm(_NormBase):\n",
    "    def __init__(self,num_features, eps=1e-5,gamma=1., beta=0.):\n",
    "        super(MyLayerNorm, self).__init__(num_features,eps)\n",
    "        self.gamma = gamma\n",
    "        self.beta = beta\n",
    "\n",
    "    def forward(self, input: Tensor) -> Tensor:\n",
    "        # input_shape:[B, C, H, W]\n",
    "        B,C,H,W = input.shape\n",
    "        if C != self.num_features:\n",
    "            print(\"num_features don't match.\")\n",
    "            return\n",
    "        x_mean = torch.mean(input,(1,2,3),True)\n",
    "        x_var = torch.var(input,(1,2,3),keepdim=True)\n",
    "        x_normalized = (input - x_mean) / torch.sqrt(x_var + self.eps)\n",
    "        results = self.gamma * x_normalized + self.beta\n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "9bf30a15",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[[ 3.8814e-01,  2.8692e-01, -1.4588e+00, -1.4415e+00,  1.4297e+00],\n",
      "          [ 3.5429e-01,  1.6277e+00, -1.4778e+00, -2.9158e-01,  1.2217e+00],\n",
      "          [ 2.4688e-01, -1.3306e+00, -1.9646e+00, -2.1270e+00, -3.0229e-01],\n",
      "          [ 5.3620e-01, -5.3309e-01,  2.0545e-01, -1.3918e-01,  5.0365e-01],\n",
      "          [-1.6655e+00,  1.3526e-01,  6.0800e-01, -5.2397e-01,  7.3469e-01]],\n",
      "\n",
      "         [[-6.8194e-02, -2.1238e+00,  2.2362e+00, -3.9910e-01,  9.4643e-01],\n",
      "          [-1.4434e-01, -1.1685e+00, -2.4830e-01, -8.4777e-01, -1.9725e+00],\n",
      "          [-1.4047e+00, -1.3617e+00,  8.8774e-01,  2.6046e-01, -1.8201e-01],\n",
      "          [ 1.2042e+00, -5.9070e-01, -3.4294e-01, -5.4540e-02, -6.6649e-01],\n",
      "          [ 1.2993e+00, -8.1122e-01,  6.4138e-01,  4.0110e-01, -1.1704e+00]],\n",
      "\n",
      "         [[ 2.2184e+00, -2.3412e-01, -8.5701e-02, -7.3325e-01, -6.1114e-01],\n",
      "          [ 9.1601e-01,  9.0128e-01,  8.0706e-01, -2.3807e-01, -8.2848e-01],\n",
      "          [ 4.7207e-01, -9.9560e-01, -6.7728e-01, -1.9831e-01,  1.6806e+00],\n",
      "          [ 1.1354e+00,  1.2637e+00, -2.1259e-04,  7.7790e-01, -2.9395e-01],\n",
      "          [ 7.0622e-01,  6.4234e-01, -5.2342e-01,  7.3368e-01, -9.7658e-01]],\n",
      "\n",
      "         [[-3.3294e-02, -4.0143e-01, -3.6582e-01, -1.3411e+00,  8.7730e-01],\n",
      "          [ 3.0073e-01,  2.5834e+00,  1.5622e-02, -1.4562e+00, -9.7832e-02],\n",
      "          [-8.0706e-01,  2.3334e-01, -1.4408e-01, -1.4015e-01, -3.0168e-01],\n",
      "          [ 7.2120e-01, -3.7501e-01,  1.9763e+00,  1.1584e+00,  1.6368e+00],\n",
      "          [-1.0387e-01,  3.1922e-02, -4.7342e-01,  1.6238e-02,  1.2888e+00]]],\n",
      "\n",
      "\n",
      "        [[[ 9.1381e-01,  9.5245e-01, -5.9332e-01, -3.1157e-01, -1.7462e-01],\n",
      "          [-1.2957e+00, -7.1610e-02,  1.6812e+00, -3.7919e-01, -1.3250e+00],\n",
      "          [-1.1418e-01,  6.0998e-01, -1.9998e+00,  7.1587e-01,  2.6410e+00],\n",
      "          [ 3.5681e-01, -2.4463e-01, -3.9791e-01, -1.5076e+00, -1.5005e+00],\n",
      "          [ 1.0092e+00, -1.4062e+00, -2.4191e-01, -1.2153e+00,  2.2057e+00]],\n",
      "\n",
      "         [[-3.2669e-01, -1.4504e-02,  6.4265e-01,  1.2388e+00,  6.3814e-01],\n",
      "          [ 2.0583e-01, -1.3536e+00, -8.2438e-01,  1.4948e-01,  1.1530e+00],\n",
      "          [ 6.3626e-02, -1.1843e+00,  2.5660e-01,  3.2118e-01, -9.7542e-01],\n",
      "          [-4.3198e-01, -1.3926e+00,  9.7831e-01, -4.2294e-01, -7.8418e-01],\n",
      "          [-2.1372e+00, -2.7055e-01,  1.6864e-01,  4.8139e-01, -6.3993e-01]],\n",
      "\n",
      "         [[ 9.4995e-01,  1.4497e+00,  1.0734e+00, -1.0652e+00, -2.4391e-01],\n",
      "          [-2.4455e-01,  1.4974e+00,  5.6739e-02,  1.4176e-01, -8.6017e-01],\n",
      "          [-1.6362e-01,  4.7042e-01,  2.6092e-01, -9.9103e-01,  1.0327e+00],\n",
      "          [ 5.4472e-01, -5.5808e-01, -3.1003e-01,  1.8748e+00, -2.0624e-01],\n",
      "          [ 2.1129e+00, -1.0160e+00, -6.3533e-01,  1.4182e+00,  4.7948e-01]],\n",
      "\n",
      "         [[ 5.4894e-01, -2.1658e-01, -2.5495e-01,  2.4458e+00,  1.2314e+00],\n",
      "          [ 1.1860e+00, -8.1402e-01, -1.3419e+00,  1.2695e-01,  1.9296e-01],\n",
      "          [-4.0855e-01, -2.1669e+00,  8.2375e-01,  3.8716e-01, -2.4987e-01],\n",
      "          [ 6.8232e-01, -8.1211e-01, -7.4757e-01, -1.8608e+00,  1.3282e-01],\n",
      "          [-2.5540e-01, -5.0522e-02,  2.0332e-03,  1.5977e-01,  3.4394e-01]]]])\n"
     ]
    }
   ],
   "source": [
    "m2 = MyLayerNorm(4)\n",
    "output2 = m2(input)\n",
    "print(output2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fe1f530",
   "metadata": {},
   "source": [
    "### （3）InstanceNorm实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "2e0271e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyInstanceNorm(_NormBase):\n",
    "    def __init__(self,num_features, eps=1e-5,gamma=1., beta=0.):\n",
    "        super(MyInstanceNorm, self).__init__(num_features,eps)\n",
    "        self.gamma = gamma\n",
    "        self.beta = beta\n",
    "\n",
    "    def forward(self, input: Tensor) -> Tensor:\n",
    "        # input_shape:[B, C, H, W]\n",
    "        B,C,H,W = input.shape\n",
    "        if C != self.num_features:\n",
    "            print(\"num_features don't match.\")\n",
    "            return\n",
    "        x_mean = torch.mean(input,(2,3),True)\n",
    "        x_var = torch.var(input,(2,3),keepdim=True)\n",
    "        x_normalized = (input - x_mean) / torch.sqrt(x_var + self.eps)\n",
    "        results = self.gamma * x_normalized + self.beta\n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "c602de8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[[ 0.5516,  0.4565, -1.1833, -1.1671,  1.5301],\n",
      "          [ 0.5198,  1.7160, -1.2012, -0.0869,  1.3346],\n",
      "          [ 0.4189, -1.0629, -1.6585, -1.8110, -0.0969],\n",
      "          [ 0.6907, -0.3137,  0.3800,  0.0563,  0.6601],\n",
      "          [-1.3775,  0.3141,  0.7582, -0.3052,  0.8772]],\n",
      "\n",
      "         [[ 0.1507, -1.7979,  2.3353, -0.1629,  1.1126],\n",
      "          [ 0.0786, -0.8923, -0.0200, -0.5883, -1.6545],\n",
      "          [-1.1162, -1.0755,  1.0569,  0.4623,  0.0428],\n",
      "          [ 1.3569, -0.3446, -0.1097,  0.1637, -0.4164],\n",
      "          [ 1.4471, -0.5536,  0.8234,  0.5956, -0.8941]],\n",
      "\n",
      "         [[ 2.2685, -0.5356, -0.3659, -1.1063, -0.9667],\n",
      "          [ 0.7794,  0.7625,  0.6548, -0.5401, -1.2152],\n",
      "          [ 0.2718, -1.4063, -1.0423, -0.4947,  1.6536],\n",
      "          [ 1.0302,  1.1769, -0.2682,  0.6215, -0.6040],\n",
      "          [ 0.5395,  0.4665, -0.8664,  0.5709, -1.3845]],\n",
      "\n",
      "         [[-0.2344, -0.6175, -0.5805, -1.5955,  0.7132],\n",
      "          [ 0.1132,  2.4888, -0.1835, -1.7152, -0.3016],\n",
      "          [-1.0397,  0.0431, -0.3497, -0.3456, -0.5137],\n",
      "          [ 0.5508, -0.5900,  1.8570,  1.0058,  1.5036],\n",
      "          [-0.3079, -0.1666, -0.6925, -0.1829,  1.1414]]],\n",
      "\n",
      "\n",
      "        [[[ 0.8168,  0.8490, -0.4374, -0.2029, -0.0890],\n",
      "          [-1.0219, -0.0032,  1.4554, -0.2592, -1.0463],\n",
      "          [-0.0387,  0.5640, -1.6079,  0.6521,  2.2542],\n",
      "          [ 0.3533, -0.1472, -0.2748, -1.1983, -1.1923],\n",
      "          [ 0.8962, -1.1139, -0.1450, -0.9550,  1.8919]],\n",
      "\n",
      "         [[-0.1747,  0.1931,  0.9675,  1.6699,  0.9621],\n",
      "          [ 0.4528, -1.3847, -0.7611,  0.3864,  1.5689],\n",
      "          [ 0.2852, -1.1853,  0.5126,  0.5887, -0.9391],\n",
      "          [-0.2988, -1.4306,  1.3630, -0.2881, -0.7137],\n",
      "          [-2.3080, -0.1086,  0.4089,  0.7774, -0.5438]],\n",
      "\n",
      "         [[ 0.7101,  1.2420,  0.8416, -1.4347, -0.5606],\n",
      "          [-0.5612,  1.2928, -0.2406, -0.1501, -1.2165],\n",
      "          [-0.4751,  0.1997, -0.0232, -1.3558,  0.7982],\n",
      "          [ 0.2788, -0.8950, -0.6309,  1.6945, -0.5205],\n",
      "          [ 1.9480, -1.3824, -0.9772,  1.2085,  0.2094]],\n",
      "\n",
      "         [[ 0.5991, -0.1841, -0.2234,  2.5397,  1.2972],\n",
      "          [ 1.2508, -0.7953, -1.3354,  0.1673,  0.2349],\n",
      "          [-0.3805, -2.1794,  0.8802,  0.4335, -0.2182],\n",
      "          [ 0.7355, -0.7934, -0.7274, -1.8662,  0.1733],\n",
      "          [-0.2238, -0.0142,  0.0395,  0.2009,  0.3893]]]])\n"
     ]
    }
   ],
   "source": [
    "m3 = MyInstanceNorm(4)\n",
    "output3 = m3(input)\n",
    "print(output3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2db54f0",
   "metadata": {},
   "source": [
    "### （4）GroupNorm实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "477dfcd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyGroupNorm(_NormBase):\n",
    "    def __init__(self,g,num_features,eps=1e-5,gamma=1., beta=0.):\n",
    "        super(MyGroupNorm, self).__init__(num_features,eps)\n",
    "        self.gamma = gamma\n",
    "        self.beta = beta\n",
    "        self.group = g\n",
    "\n",
    "    def forward(self, input: Tensor) -> Tensor:\n",
    "        # input_shape:[B, C, H, W]\n",
    "        x = input.view(input.shape[0], self.group, int(input.shape[1]/self.group), input.shape[2], input.shape[3])\n",
    "        \n",
    "        x_mean = torch.mean(x,(2,3,4),True)\n",
    "        x_var = torch.var(x,(2,3,4),keepdim=True)\n",
    "        x_normalized = (x - x_mean) / torch.sqrt(x_var + self.eps)\n",
    "        results = self.gamma * x_normalized + self.beta\n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "9ac27b84",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[[[ 0.5516,  0.4565, -1.1833, -1.1671,  1.5301],\n",
      "           [ 0.5198,  1.7160, -1.2012, -0.0869,  1.3346],\n",
      "           [ 0.4189, -1.0629, -1.6585, -1.8110, -0.0969],\n",
      "           [ 0.6907, -0.3137,  0.3800,  0.0563,  0.6601],\n",
      "           [-1.3775,  0.3141,  0.7582, -0.3052,  0.8772]]],\n",
      "\n",
      "\n",
      "         [[[ 0.1507, -1.7979,  2.3353, -0.1629,  1.1126],\n",
      "           [ 0.0786, -0.8923, -0.0200, -0.5883, -1.6545],\n",
      "           [-1.1162, -1.0755,  1.0569,  0.4623,  0.0428],\n",
      "           [ 1.3569, -0.3446, -0.1097,  0.1637, -0.4164],\n",
      "           [ 1.4471, -0.5536,  0.8234,  0.5956, -0.8941]]],\n",
      "\n",
      "\n",
      "         [[[ 2.2685, -0.5356, -0.3659, -1.1063, -0.9667],\n",
      "           [ 0.7794,  0.7625,  0.6548, -0.5401, -1.2152],\n",
      "           [ 0.2718, -1.4063, -1.0423, -0.4947,  1.6536],\n",
      "           [ 1.0302,  1.1769, -0.2682,  0.6215, -0.6040],\n",
      "           [ 0.5395,  0.4665, -0.8664,  0.5709, -1.3845]]],\n",
      "\n",
      "\n",
      "         [[[-0.2344, -0.6175, -0.5805, -1.5955,  0.7132],\n",
      "           [ 0.1132,  2.4888, -0.1835, -1.7152, -0.3016],\n",
      "           [-1.0397,  0.0431, -0.3497, -0.3456, -0.5137],\n",
      "           [ 0.5508, -0.5900,  1.8570,  1.0058,  1.5036],\n",
      "           [-0.3079, -0.1666, -0.6925, -0.1829,  1.1414]]]],\n",
      "\n",
      "\n",
      "\n",
      "        [[[[ 0.8168,  0.8490, -0.4374, -0.2029, -0.0890],\n",
      "           [-1.0219, -0.0032,  1.4554, -0.2592, -1.0463],\n",
      "           [-0.0387,  0.5640, -1.6079,  0.6521,  2.2542],\n",
      "           [ 0.3533, -0.1472, -0.2748, -1.1983, -1.1923],\n",
      "           [ 0.8962, -1.1139, -0.1450, -0.9550,  1.8919]]],\n",
      "\n",
      "\n",
      "         [[[-0.1747,  0.1931,  0.9675,  1.6699,  0.9621],\n",
      "           [ 0.4528, -1.3847, -0.7611,  0.3864,  1.5689],\n",
      "           [ 0.2852, -1.1853,  0.5126,  0.5887, -0.9391],\n",
      "           [-0.2988, -1.4306,  1.3630, -0.2881, -0.7137],\n",
      "           [-2.3080, -0.1086,  0.4089,  0.7774, -0.5438]]],\n",
      "\n",
      "\n",
      "         [[[ 0.7101,  1.2420,  0.8416, -1.4347, -0.5606],\n",
      "           [-0.5612,  1.2928, -0.2406, -0.1501, -1.2165],\n",
      "           [-0.4751,  0.1997, -0.0232, -1.3558,  0.7982],\n",
      "           [ 0.2788, -0.8950, -0.6309,  1.6945, -0.5205],\n",
      "           [ 1.9480, -1.3824, -0.9772,  1.2085,  0.2094]]],\n",
      "\n",
      "\n",
      "         [[[ 0.5991, -0.1841, -0.2234,  2.5397,  1.2972],\n",
      "           [ 1.2508, -0.7953, -1.3354,  0.1673,  0.2349],\n",
      "           [-0.3805, -2.1794,  0.8802,  0.4335, -0.2182],\n",
      "           [ 0.7355, -0.7934, -0.7274, -1.8662,  0.1733],\n",
      "           [-0.2238, -0.0142,  0.0395,  0.2009,  0.3893]]]]])\n"
     ]
    }
   ],
   "source": [
    "m4 = MyGroupNorm(2,4)\n",
    "output4 = m4(input)\n",
    "print(output4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cea1a452",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch]",
   "language": "python",
   "name": "conda-env-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
